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(54) Video camera surveillance system detects 
intruders and/or fire 

(57) A plurality of colour video cameras 10 provides video 
signals which are switched 18, under the control of 
computer 32 to signal processor 1 00 having a frame 
memory 22 and means for recognizing the presence of an 
intruder 28 or a fire 26 respectively; mode switch allows for 
the stored video signals to be subject to either fire or 
intruder detection or both. If an abnormality is detected in 
the output of one video camera, the video signal of that 
camera is routed to monitor 42 for display. Fire is detected 
(fig 2) by comparing the radiant energy, calculated from the 
temperature (ratio G:R or B:R), the flame outline (using a 
luminance threshold) and flame distance (using the output 
of the camera autofocus), with a threshold and observing 
changes with time in it. 
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DESCRIPTION 

SURVEILLANCE MONITOR SYSTEM USING IMAGE PROCESSING 

The present invention relates to a surveillance monitor 
system for monitoring fires and thefts, in particular one 
designed so that it can recognize and warn of both fires and 
thefts by monitoring images from the same television camera. 
The invention also relates to a surveillance monitor system 
which detects a fire by a fire detection method using images 
of a surveillance area, from which the system deduces the 
radiant energy of the radiant energy source and, from this 
data, judges whether there is a fire. 

There are known fire monitor systems using television 
cameras and having the virtue of providing a great quantity 
of information and monitoring a large surveillance area. Also 
there are known burglary monitor systems using television 
cameras . 

An example of a fire monitor system using a television 
camera is Japanese Patent Laid Open 1-268570, a fire 
extinguishing system . 

This system can recognize the size of a flame and detect 
the position of a flame by processing luminance signals of a 
monitor image obtained by a monitor television camera. In 
more detail, it compares an image's luminance signals with a 
threshold value, and judges a part of the image whose 
luminance signals exceed the threshold value to be a fire 
source. As an example, this method for detecting a radiant 
energy source based on luminance signals is used for 
detecting a radiant energy source for the purpose of 
controlling an extinguishing system in Japanese Patent Laid 
Open 1-268572'. If there are a plurality of flames it 
extinguishes the fire by controlling the nozzle based on 
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arranging the targets to be extinguished in order of size 
from the largest flame downwards. 

Again, an example of an ant i -burglary monitor system 
using a television camera is Japanese Patent Laid Open 2- 
171897, an abnormality monitor system. In this system a 
reference image of the surveillance area in v its normal state 
is memorized beforehand, and edges in a current image 
obtained by the television camera are sampled and compared 
with the reference image. Then only (an edge or) edges which 
appear/ (s) in the current image but not in the reference 
image are/ (is) sampled, and if the number of pixels of the 
area surrounded by the edge exceeds the threshold value 
number, it recognizes it as an intruder. Also the new edge 
information of current images is constantly logically added 
to amend the reference image so that, even though the shadows 
of objects in the room move with the movement of the sun, 
t h eS e shadows are not mistakenly recognized as intruders. 

However, known monitor systems using television cameras 

r ^ 

were divided into systems monitoring fires and systems .or 
an~i -burglary monitoring, and thought of separately, so that 
even if they were installed in the same building, they were 
installed as separate systems and also the data irom tne two 
systems was processed separately, resulting in the following 
problems . 

Firstly, compared with monitor systems using presently 
used fire sensors and intruder detectors the system cost of a 
monitor system using a television camera was rather high 
because an image processor exclusively for processing the 
enormous quantity of data was used. As a result surveillance 
monitor systems using a television camera are not in 
widespread use. 

Secondly, the image processing technology for recognizing 
fires and for recognizing intruders had not been perfected or 
brought to maturity, and therefore there was still the 
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an image-taking device for taking an image of 
areascape of a surveillance area; 

a fire image recognition section for recognizing 
a fire from a monitor image taken by said image-taking 
device and outputting fire detection information ; 

an intruder entity image recognition section for 
recognizing an intruding entity from a monitor image 
taken by said image-taking device and outputting 
intruder detection information; 

a mode switching section for specifying that a 
monitor image be processed by said fire image 
recognition section and/or said intruder entity image 
recognition section; 

a pre-alarm output section for outputting a pre- 
alarm in a central monitor location upon receipt of 
detection information from said fire image recognition 
or said intruder image recognition section; and 

an image switching section for switching, in 
response to said pre-alarm, a monitor image in said 
central monitor location to an image of a location 
where an abnormality has been detected. 

A preferred surveillance monitor system of the 
present invention using image processing is provided 
with image-taking devices and fire image recognizer 
which recognizes a fire from an image taken by the 
image-taking devices and outputs fire detection 
information, and an intruder image recognizer which 
recognizes an intruder from the images taken by the 
image-taking devices and outputs burglary information, 
and a mode switcher which specifies that image 
processing be of images from fire image recognizer 
and/or intruder image recognizer, and an image 
switcher which, when a pre-alarm is outputted, 
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switches the monitor image in the central monitor 
location monitor to the image of the location at which 
an abnormality has been detected. 

As a structure for performing image processing in 
a surveillance monitor system according to the present 
invention, the following can be provided: image-taking 
means for taking images of the surveillance area; a 
flame outline sampler which samples the outlines of 
flames included in images obtained by the image-taking 
means; a temperature detection means which detect the 
distribution temperature of the flame outline sampled 
by flame outline sampler; a distance determining means 
which determines the distance to the fire source; a 
radiant energy calculator which calculates the radiant 
energy E from the fire source, based on (1) the 
distribution temperature inside the flame outline, 
sampled by the flame outline sampler and (2) the 
distance to the flame source; and which judges whether 
there is a fire or not, using the fire source's energy 
E found by the radiant energy calculator. 

The image-taking means takes colour images, and 
from the ratio of the G component to the R component 
(G/R) or of the B component to the R component (B/R) 
in these images the temperature detection means find, 
for each pixel, the surface temperature of the flame, 
by referring to a conversion table for obtaining 
distribution temperatures . 
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As another example of a temperature detection means, the 
distribution temperature of images may be detected based on 
the measurement output of a radiation thermometer. 

Further, the fire judging means ran judge whether there is 
a fire or not based on the radiant energy obtained in each 
prescribed sampling period and the change in radiant energy 
between sampling times. 

In addition, the distance detection means can establish 
the distance to the radiant energy source by using the amount 
of movement of the lens of the image taking means' autofocus 
mechanism or by using a distance finder. 

In a surveillance monitor system of the present 
invention using image processing and of the structure 
described above, one television camera is provided for each 
surveillance area, and then from its monitor images, fire 
and/or burglary can be recognized by image processing of fire 
and/or intruder entity images, according to the mode setting. 

As a result, the image processing hardware, namely 
television cameras, ADC and frame memory, can be used in 
common for both fire monitoring and burglary monitoring. In 
fact, all that is required is to load the fire image 
recognition software and the intruder recognition software. 
Therefore, as the system structure is simpler than with 
separate systems and a large reduction in cost can be 
achieved, more widespread use of surveillance monitor systems 
using television cameras can be promoted. 

Further, even if the technology for recognition of fire 
and intruders are not necessarily perfect, if a fire or 
burglary is detected by image processing, a pre-alarm is 
outputted in the central monitor room and the monitor image 
is manually or automatically switched to the image of the 
location where the abnormality has been detected. 

Therefore, on receiving the pre-alarm output a 
surveillance person can verify whether there is a fire, or a 



burglary, by checking the image of the location, and will 
then immediately know whether there is an abnormality such as 
a fire or whether it is a false alarm. Moreover, even if 
there is an image processing misrecognition, the reliability 
of the system can be sufficiently maintained despite 
producing any false report, since a human being sees the 
television camera image of the actual location and makes the 
final judgement. 

In addition, a surveillance monitor system of the 
invention performing fire detection by an image-using fire 
detection method having such a structure samples as a flame 
outline any outline in an image taken by the image -taking 
camera whose luminance signals exceed a predetermined level 
and finds the distribution temperature inside the sampled 
outline. If the outline shape and distribution temperature 
are thus found, the radiant energy E from inside the flame 
outline can be calculated based on the Stefan- Boltzmann law. 

Further, from the fact that the size of a flame on the 
screen varies depending on the distance, the area of the 
flame on the screen can be revised to the true flame area by 
measuring the distance to the fire source, and the amount cf 
energy being radiated ( amount of heat produced ) from the 
flames can be estimated. 

Thus it is possible, if the radiant energy from the 
flames can be estimated, zo judge whether it is a fire or 
some other energy source. Also, by looking at the changes, 
i.e. transitions, in radiant energy of each sampling 
combined, them it can judge accurately whether there is a 
fire' or not . 

That is to say, because it judges whether there is a 
fire or not by estimating the radiant energy taking the 
portion which has a constant luminance level as an 
abnormality, it can perform fire judgement with a high degree 
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of reliability unaffected by the size or shape of the 
surveillance area. 

Also, because it judges a fire by directly detecting the 
radiant energy from flames, it can judge a f ire in a shorter 
. time than known f ire detectors which use smoke and/or heat. 

In addition, since it can judge a fire from monitor 
images from a camera, the area which can be covered by one 
system is increased, and if a surveillance area scanning 
mechanism and/or zoom mechanism is provided on the camera an 
even larger area can be monitored. 

The invention is nnw fiomi-i h^H further, hy way of 
example only, with reference to the accompanying drawings, 
in which: 

Fig.l is a block diagram showing one embodiment of the 
present invention; 

Fig. 2 is a block diagram showing the section relatinc to 
fire detection of the image processing unit of the first " 
embodiment of the invention; 

Fig. 3 is a graph showing the characteristic used as a 
3513 f ° r CORVersion of G/R ratio into distribution 
temperature ( used for temperature detection ) in the H rst 
embodiment of the invention; 

Fig. 4 is a diagram showing sampling of a flame outline 
in the first embodiment of the invention; 

Fig. 5 is a block diagram showing the structure of a 
second embodiment of the invention- and 

Fig. 6 is a graph showing the characteristic used as a 
basis' for conversion of B/R ratio into distribution 
temperature ( used for temperature detection ) in the firsr 
embodiment of the invention. 

Fig.l is a block diagram showing the structure of a 
first embodiment according to the present invention Fig 2 
shows the structure of the. subsections of Fia . 1 ' s image 
processing section relating to fire detection. In Fig.l 10-1, 
10-2, ...io-n are television cameras acting as image-taking 
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cevices. In this embodiment one television camera is provided 
in each of surveillance areas 12-1, 12-2, . ..12-n, for 
example one television camera on each floor of a building. As 
television cameras 10-l~10n,for example color CCD cameras or 
infrared CCD cameras, or . any other suitable image-taking 
device may be used. 

In this embodiment, CCD cameras are used as image -taking 
means and autofocusses 112 are provided, and, for example, 
each one is placed in such a position that it can cover the 
whole of each surveillance area. Also, depending on 
requirements, the television cameras 10 may be made to scan 
the surveillance area. 

Television cameras 10-l~10n each take an image of the 
surveillance area once in each determined sampling period, 
and the color images thus obtained are passed to image 
processing section 16. 

Image processing section 16 comprises camera switcher 18 
as image switcher, analog - digital converter ( hereinafter ADC 
) 20 and image processor 100. This image processor 100 has a 
frame memory 22, a mode switcher 24 , a fire image 
recognizer 26, an intruder image recognizer 28 and an 
abnormality judger 30. 

The structure of image processing section 16 will now be 
described, together with its operation, in detail. Camera 
switcher 18 switches television cameras 1 0 - 1 ~ 1 0 - n once in 
each fixed switching cycle and sends each monitor image to 
ADC 20. This ADC 20 samples, in a fixed sampling cycle, 
signals obtained as picture signals, for example RGB R, G and 
B component signals and luminance signals, on a pixel basis 
and then converts them into multitonal digital data. Then 
the data converted by ADC 20 from the frame obtained from 
television camera 10 is stored in frame memory 22. That is, 
RGB and luminance signals are stored in R component frame 
memory 116, G component frame memory 118, 3 component frame 
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memory 120 and luminance component frame memory 126 
respectively. The capacity of frame memory 22 is decided 
appropriately based on the number of frames needed for image 
recognition. 

Based on mode switch signals from outside, mode switcher 
24, which is disposed subsequent to frame memory 22, reads 
out from frame memory mode switcher 24 image data to be 
processed. Then it sends them to the specified image 
recognizer or recognizers, fire image recognizer 26 and/or 
intruder recognizer 28. 

There are three types of mode between which mode 
switcher 24 switches. They are: 

(a) fire image recognition mode; 

(b) intruder entity image recognition mode; 

(c) fire and intruder entity image recognition 
mode . 

Concretely, in this case the way of using these modes might 
be to specify for mode switcher 24, for example mode (a) 
( i.e. fire image recognition mode ) for the daytime and mode 
(c) (i.e. fire and intruder entity image recognition mode ) 
for the nighttime. 

Fire image recognizer 25 samples fire-caused images from 
the pictures obtained by the television camera and performs 
recognition on them. As a means for doing this there is a 
method, e.g. as in Japanese Patent Laid Open 1-268570, of 
sampling a fire-caused flame portion from the luminance 
signals in the monitor images and performing recognition. At 
this time, based on the recognition processing using the 
sampling of this flame portion, the size of the flame region 
may be recognized in addition. Also, fire may be recognized 
from the flicker frequency of a flame detected from luminance 
signals. Again, fire may be recognized from smoke density or 
speed of ascent of the fire's smoke, capturing by image 
processing the smoke from the fire captured in monitor 
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irnages. Other suitable processing methods for fire 
recognition by monitor image processing can be used without 
any modification. 

In regard to the above, details of the structure of the 
present invention are as follows. Fire image recognizer 26 
comprises a ratio calculator 122, a temperature converter 
124, a flame outline sampler 128, a distance detection device 
130 and a radiant energy calculator 132. 

Ratio calculator 122 and temperature converter 124 are 
provided as temperature detection means which detects the 
distribution temperature of images from television camera 10. 
That is, ratio calculator 122 calculates the G/R of a G 
component and an R component ( of the same pixel position ) 
obtained by reading out G component frame memory 118 and R 
component frame memory 116. By using a conversion table based 
on an experimentally obtained characteristic (shown in Fig. 3), 
and by accessing the conversion table with the G/R ratio 
value obtained by ratio calculator 122 the distribution 
temperature for each pixel can be found based on the 
characteristic of Fig . 3 . Also, the distribution temperature 
for each pixel may be found from the B/R ratio by calculating 
5/R. the ratio of the B component to the R component, and 
using a conversion table based cn experimentally obtained 
characteristics shown in Fig . 6 . 

Flame outline sampler 128 is disposed subsequent to 
luminance component frame memory 126. Among the luminance 
signals stored in luminance component frame memory 126, flame 
outline sampler 128 samples as flame outlines the outlines of 
regions which exceed a prescribed level. The signals showing 
the amount of movement of the image- taking lens caused by 
autofocus control by television camera 10' s autofocus 112 can 
be passed to distance detection device 130 and the distance 
to the in- focus position, e.g. the distance to flame 114, can 
be calculated. 
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The temperature found by temperature converter 124, the 
outline information obtained by flame outline sampler 128, 
and the distance to flame 114 detected by distance detection 
device 130 are passed to radiant energy calculator 132 and 
calculation to estimate the radiant energy inside the flame 
outline is performed. 

The calculation performed by radiant energy calculator 
132 is based on the Stefan- Boltzmann law. The temperature T 
of the flame is given by the following equation: 

E = ex(JxSxT 4 [ watt] (1) 

But O- Stefan — Boltzmam constant = 5 . 67x10"' [W- m 2 • k M ] 
S: surface area of flame [m A ] 
e: radiation rate. 
Here radiation rate £ and a are treated as constants. For 
the radiation rate it was found by experiment that a value of 
approximately 0.2 may be used. 

Fig. 4 shows one example of a flame outline 148 sampled 
by flame outline sampler 128 from an image of luminance 
component frame memory 126. For pixels of the diagonally 
hatched area inside the flame outline 148, convolution 
integration of ( Si x Ti* ) , in which Si is the area of 1 
pixel and Ti is the temperature of each pixel, can be carried 
out, by using the temperature obtained by temperature 
converter 124. Concretely, 

r. 

E = £• a-Z Si . Ti 4 [watt] (2) 

i- 1 

may be used, n is the number of pixels inside the flame 
outline. 

' Further, the area S of an individual pixel, used in the 
calculation of equation (2) , is converted into the area on an 
actual flame by using the detected distance from distance 
detection device 130. Note that here the area of the flame is 
for the flame considered as a plane. 
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In detail, according to the present embodiment, fire 
judgement is performed as described next. Referring again to 
Fig. 2, the radiant energy inside the flame outline calculated 
by radiant energy calculator 132 is passed to fire judger 
30a, which judges whether there is a fire or not. Fire judger 
30a is provided with No . 1 judger 136, which judges whether 
there is a fire or not from the radiant energy of each 
sampling, and No . 2 judger 138, which judges whether there is 
a fire or not from change in radiant energy between sampling 
times, that is to say from radiant energy transition. For 
example, No.l fire judger 136 has two threshold levels, a 
higher one and a lower one, and if the radiant energy exceeds 
the higher threshold level then it immediately judges that 
there is a fire. On the other hand, if the radiant energy 
exceeds the lower but not the higher threshold level, change 
in radiant energy between sampling times is checked by No . 2 
judger 138. Then, if, in addition to there being an increase 
in radiant energy, the rate of increase of radiant energy 
exceeds a prescribed value, it judges that there is a fire. 
Further, even if the radiant energy exceeds the higher 
threshold level in No.l fire judger 136 and No.l fire judger 
136 judges than there is a fire, the change in radiant energy 
between sampling times is checked by No . 2 fire judger 138, 
and if there "is almost no change, i.e. if the radiant energy 
is constant, it judges that there is a steady flame, a flame 
other than a fire, in the surveillance area, and therefore 
it does not judge that there is a fire. 

With regard to the recognition results from intruder 
image recognizer 28, in the same way it performs a comparison 
between a threshold value and the number of pixels inside an 
edge sampled as a human form, and judges whether there is an 
intruder from the movement over time of an edge recognized as 
a human form. 
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send the command to display the image of the location to the 
computer itself 44, which sends a switching control signal to 
a camera switcher 18 of image processing section 16. Thus, 
for example, even if at this time a television camera of a 
surveillance area other than the television camera of the 
surveillance area, 12-1, in which a fire has been detected, 
had been switched to, television camera 10-1 of surveillance 
area 12-1 where the fire has been recognized will be 
compulsorily switched to. Then pictures of the location where 
an abnormality has been recognized will be transferred onto 
the monitor display 42 via ADC 20 and frame memory 22. Of 
course, the switching command need not be given by the 
surveillance person operating keyboard 46, but may be given 
by sending a control signal automatically to camera switcher 
18 simultaneously with the outputting of a pre -alarm, based 
on receipt by host computer 32 of abnormality detection 
information . 

Next, when monitor display 42 has been switched to the 
abnormality location pictures by manual operation by a 
surveillance person or automatically, the surveillance person 
looks at the location pictures and if he confirms that there 
is a fire abnormality or a burglary abnormality, he for 
example performs confirmation input: operation using the 
keyboard 46. As a result, alarm panel 34 receives an alarm 
command signal from the computer itself 44, and an actual 
alarm is issued. At the same time, in the case of a fire, a 
gang control signal is issued to control panel 36 to gang 
control smoke ducts, fire shutters and other anti-fire and 
antrl - smoke equipment . 

Again, if a burglary is detected automatic reporting to 
the police and/or other outside organizations is performed by 
an automatic notifier not shown in the illustrations. 

It should be noted that the switching orders for mode 
switcher 24 of image processing section 16 come from host 
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CLAIMS 

1. A surveillance monitor system using image processing, 
comprising : 

an image- taking device for taking an image of 
areascape of a surveillance area; 

a fire image recognition section for recognizing a 
fire from a monitor image taken by said image -taking device 
and outputting fire detection information; 

an intruder entity image recognition section for 
recognizing an intruding entity from "a monitor image taken by 
said image -taking device and outputting intruder detection 
information; 

a mode switching section for specifying that a 
monitor image be processed by said fire image recognition 
section and/or said intruder entity image recognition 
section; 

a pre- alarm output section for outputting a pre- 
alarm in a central monitor location upon receipt of detection 
information from said fire image recognition or said intruder 
image recognition section; and 

an image switching section for switching, in 
response to said pre-alarm, a monitor image in said central 
monitor location tc an image of a location where an abnormality 
has been detected. 

2; A surveillance monitor system according to Claim 1, 

wherein said fire image recognition means comprises fire 

judging means using image processing for judging whether or 

not there is a fire, based on a radiant energy source's 

radiant energy established by a radiant energy calculating means, 

which fire judging means comprises: 

flame outline sampling means for sampling as an 
outline of a flame an area whose luminance signals, included 
in an image taken by said image -taking means, exceed a 
prescribed level; 



-19- 

at least temperature detection means for detecting a 
distribution temperature inside a flame outline sampled by said 
flame region outline sampling means; 

distance measurement means for measuring the distance to a 
radiant energy source; and 

radiant energy calculating means for calculating radiant 
energy from the radiant energy source, based on the distribution 
temperature inside the flame outline sampled by said flame 
outline sampling means. 

3. A surveillance monitor system according to claim 2, 
wherein said image-taking means takes colour images, and said 
temperature detection means finds for each pixel a flame surface 
temperature using a ratio of the G component to the R component 
(G/R) or a ratio of the B component to the R component (B/R) of 
said colour images taken by said image taking means and 
referring to a conversion table for converting to distribution 
temperature. 

4. A surveillance monitor system according to claim 2, 
wherein said fire judging means judges whether or not there is a 
fire f based on radiant energy obtained in each prescribed 
sampling cycle and on a change in radiant energy between 
sampling times. 

5. A surveillance monitor system according to ; claim 2, 
wherein said temperature detecting means detects a distribution 
temperature of an image based on a measurement output of a 
radiation thermometer. 

6. A surveillance monitor system according to claim 2, 
wherein said distance detection means measures the distance to a 
radiant energy source by the amount of lens movement of an 
autofocus of said image-taking means or by using a distance 
finder . 

7. A surveillance monitor system substantially as 
hereinbefore described with reference to and as illustrated in 
the accompanying drawings . 



THIS PAGE BLANK iuspto) 



- ao 

Patents Act 1977 

^ aminer's report to the Comptroller under 
bection 17 (The Search Report) 



Application number 
GB 9214656. 2 



Relevant Technical fields 

K 

(i) UK CI {Edition ) 



<ii) Int CI (Edition ) 



G4N NFFX, H4F FAA FDG FGG 
G08B 13/194 13/196 19/00 



Databases (see over) 
(i) UK Patent Office 

ONLINE DATABASE : WPI 

(ii) 



Documents considered relevant following a search in respect of claims 



I Search Examiner 



M K REES 



Date of Search 



29 SEPTEMBER 1992 



1-7 



Category 
(see over) 



Identity of document and relevant passages 



GB 2183378 A MATSUSHITA see page 3, 

lines 45-52 

EP 0261917 A2 SONY see Figure 1; column 4 

line 25 - column 7 line 8 



US 4777526 



SONY see whole document 



Relevant to 
claim(s) 





SF2(p) 



GEM - doc99\ti!000228 



Category 



- zv- 

Identity of document and relevant passages 



Relfc .t 
to claim(s) 



Categories of documents 

X: Document indicating lack of novelty or of 
inventive step. 

Y: Document indicating lack of inventive step if 
combined with one or more other documents of the 
same category. 

A: Document indicating technological background 
and/or state of the art. 



P: Document published on or after the declared 
priority date but before the filing date of the 
present application. 

E: Patent document published on or after, but with 
priority date earlier than, the filing date of the 
present application. 

8c Member of the same patent family, 
corresponding document 



Databases: The UK Patent Office database comprises classified collections of GB. EP, WO and US 
patent specifications as outlined periodically in the Official Journal (Patents). The on-line databases 
considered for search are also listed periodically in the Official Journal (Patents). 

t^. ^ b " sl L cd Il 1 ? 9 2 a k t , Thc P?*" 1 ° mcc - Concept House. CardlfTRoad. Newport. Gwent NP9 1RH. Further copies may be obtained from 
Sales Branch. Unit 6. Nine Mile Point. CwmTellnfach. Cross Keys. Newport. NP1 7HZ Printed by Multiplex techniques ltd. St Mary Cray. Kent. 



